Face masks influence emotion judgments of facial expressions: a drift-diffusion model.

Face masks slow the spread of SARS-CoV-2, but it has been unknown how masks might reshape social interaction. One important possibility is that masks may influence how individuals communicate emotion through facial expressions. Here, we clarify to what extent-and how-masks influence facial emotion communication, through drift-diffusion modeling (DDM). Over two independent pre-registered studies, conducted three and 6 months into the COVID-19 pandemic, online participants judged expressions of 6 emotions (anger, disgust, fear, happiness, sadness, surprise) with the lower or upper face "masked" or unmasked. Participants in Study 1 (N = 228) correctly identified expressions above chance with lower face masks. However, they were less likely-and slower-to correctly identify these expressions relative to without masks, and they accumulated evidence for emotion more slowly-via decreased drift rate in DDM. This pattern replicated and intensified 3 months later in Study 2 (N = 264). These findings highlight how effectively individuals still communicate with masks, but also explain why they can experience difficulties communicating when masked. By revealing evidence accumulation as the underlying mechanism, this work suggests that time-sensitive situations may risk miscommunication with masks. This research could inform critical interventions to promote continued mask wearing as needed.

Masking our emotions: Emotion recognition and perceived intensity differ by race and use of medical masks.

Although medical masks have played a key role in decreasing the transmission of communicable disease, they simultaneously reduce the availability of nonverbal cues fundamental to social interaction. In the present study, we determined the collective impact of medical masks on emotional expression recognition and perceived intensity as a function of actor race. Participants completed an emotional expression recognition task involving stimuli with or without medical masks. Across six basic emotional facial expressions, medical masks were associated with significantly more emotional expression recognition errors. Overall, the effects associated with race varied depending on the emotion and appearance of masks. Whereas recognition accuracy was higher for White relative to Black actors for anger and sadness, the opposite pattern was observed for disgust. Medical mask-wearing exacerbated actor-race related recognition differences for anger and surprise, but attenuated these differences for fear. Emotional expression intensity ratings were significantly reduced for all emotions except fear, where masks were associated with increased perceived intensity. Masks further increased already higher intensity ratings for anger in Black versus White actors. In contrast, masks eliminated the tendency to give higher intensity ratings for Black versus White sad and happy facial expressions. Overall, our results suggest that the interaction between actor race and mask wearing status with respect to emotional expression judgements is complex, varying by emotion in both direction and degree. We consider the implications of these results particularly in the context of emotionally charged social contexts, such as in conflict, healthcare, and policing.

The Eyes Have It: Psychotherapy in the Era of Masks.

Nonverbal communication is integral to the success of psychotherapy and facial expression is an important component of nonverbal communication. The SARS CoV-2 pandemic has caused alterations in how psychotherapy services are provided. In this paper, potential issues that may arise from conducting psychotherapy when both the patient and therapist are wearing masks are explored. These include higher likelihood of misidentifying facial expression, especially when expression is incongruent with body language, and when the lower face is more important for correct identification of emotion. These issues may be particularly problematic for patient populations for whom emotion recognition may be a problem at baseline, or for those more prone to biases in emotional recognition. Suggestions are made for therapists to consider when seeing patients in-person when masks are necessary.

An experimental study of animating-based facial image manipulation in online class environments.

Recent advances in artificial intelligence technology have significantly improved facial image manipulation, which is known as Deepfake. Facial image manipulation synthesizes or replaces a region of the face in an image with that of another face. The techniques for facial image manipulation are classified into four categories: (1) entire face synthesis, (2) identity swap, (3) attribute manipulation, and (4) expression swap. Out of them, we focus on expression swap because it effectively manipulates only the expression of the face in the images or videos without creating or replacing the entire face, having advantages for the real-time application. In this study, we propose an evaluation framework of the expression swap models targeting the real-time online class environments. For this, we define three kinds of scenarios according to the portion of the face in the entire image considering actual online class situations: (1) attendance check (Scenario 1), (2) presentation (Scenario 2), and (3) examination (Scenario 3). Considering the manipulation on the online class environments, the framework receives a single source image and a target video and generates the video that manipulates a face of the target video to that in the source image. To this end, we select two models that satisfy the conditions required by the framework: (1) first order model and (2) GANimation. We implement these models in the framework and evaluate their performance for the defined scenarios. Through the quantitative and qualitative evaluation, we observe distinguishing properties of the used two models. Specifically, both models show acceptable results in Scenario 1, where the face occupies a large portion of the image. However, their performances are significantly degraded in Scenarios 2 and 3, where the face occupies less portion of the image; the first order model causes relatively less loss of image quality than GANimation in the result of the quantitative evaluation. In contrast, GANimation has the advantages of representing facial expression changes compared to the first order model. Finally, we devise an architecture for applying the expression swap model to the online video conferencing application in real-time. In particular, by applying the expression swap model to widely used online meeting platforms such as Zoom, Google Meet, and Microsoft Teams, we demonstrate its feasibility for real-time online classes.

Investigating the impact of disposable surgical face-masks on face identity and emotion recognition in adults with autism spectrum disorder.

With the outburst of the COVID-19 pandemic, disposable surgical face-masks (DSFMs) have been widely adopted as a preventive measure. DSFMs hide the bottom half of the face, thus making identity and emotion recognition very challenging, both in typical and atypical populations. Individuals with autism spectrum disorder (ASD) are often characterized by face processing deficits; thus, DSFMs could pose even a greater challenge for this population compared to typically development (TD) individuals. In this study, 48 ASDs of level 1 and 110 TDs underwent two tasks: (i) the Old-new face memory task, which assesses whether DSFMs affect face learning and recognition, and (ii) the Facial affect task, which explores DSFMs' effect on emotion recognition. Results from the former show that, when faces were learned without DSFMs, identity recognition of masked faces decreased for both ASDs and TDs. In contrast, when faces were first learned with DSFMs, TDs but not ASDs benefited from a "context congruence" effect, that is, faces wearing DSFMs were better recognized if learned wearing DSFMs. In addition, results from the Facial affect task show that DSFMs negatively impacted specific emotion recognition in both TDs and ASDs, although differentially between the two groups. DSFMs negatively affected disgust, happiness and sadness recognition in TDs; in contrast, ASDs performance decreased for every emotion except anger. Overall, our study demonstrates a general, although different, disruptive effect on identity and emotion recognition both in ASD and TD population.

Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces.

Owing to the availability of a wide range of emotion recognition applications in our lives, such as for mental status calculation, the demand for high-performance emotion recognition approaches remains uncertain. Nevertheless, the wearing of facial masks has been indispensable during the COVID-19 pandemic. In this study, we propose a graph-based emotion recognition method that adopts landmarks on the upper part of the face. Based on the proposed approach, several pre-processing steps were applied. After pre-processing, facial expression features need to be extracted from facial key points. The main steps of emotion recognition on masked faces include face detection by using Haar-Cascade, landmark implementation through a media-pipe face mesh model, and model training on seven emotional classes. The FER-2013 dataset was used for model training. An emotion detection model was developed for non-masked faces. Thereafter, landmarks were applied to the upper part of the face. After the detection of faces and landmark locations were extracted, we captured coordinates of emotional class landmarks and exported to a comma-separated values (csv) file. After that, model weights were transferred to the emotional classes. Finally, a landmark-based emotion recognition model for the upper facial parts was tested both on images and in real time using a web camera application. The results showed that the proposed model achieved an overall accuracy of 91.2% for seven emotional classes in the case of an image application. Image based emotion detection of the proposed model accuracy showed relatively higher results than the real-time emotion detection.

Face masks impair facial emotion recognition and induce specific emotion confusions.

Face masks are now worn frequently to reduce the spreading of the SARS-CoV-2 virus. Their health benefits are undisputable, but covering the lower half of one's face also makes it harder for others to recognize facial expressions of emotions. Three experiments were conducted to determine how strongly the recognition of different facial expressions is impaired by masks, and which emotions are confused with each other. In each experiment, participants had to recognize facial expressions of happiness, sadness, anger, surprise, fear, and disgust, as well as a neutral expression, displayed by male and female actors of the Radboud Faces Database. On half of the 168 trials, the lower part of the face was covered by a face mask. In all experiments, facial emotion recognition (FER) was about 20% worse for masked faces than for unmasked ones (68% correct vs. 88%). The impairment was largest for disgust, followed by fear, surprise, sadness, and happiness. It was not significant for anger and the neutral expression. As predicted, participants frequently confused emotions that share activation of the visible muscles in the upper half of the face. In addition, they displayed response biases in these confusions: They frequently misinterpreted disgust as anger, fear as surprise, and sadness as neutral, whereas the opposite confusions were less frequent. We conclude that face masks do indeed cause a marked impairment of FER and that a person perceived as angry, surprised, or neutral may actually be disgusted, fearful, or sad, respectively. This may lead to misunderstandings, confusions, and inadequate reactions by the perceivers.

Surgical face masks do not impair the decoding of facial expressions of negative affect more severely in older than in younger adults.

Surgical face masks reduce the spread of airborne pathogens but also disturb the flow of information between individuals. The risk of getting seriously ill after infection with SARS-COV-2 during the present COVID-19 pandemic amplifies with age, suggesting that face masks should be worn especially during face-to-face contact with and between older people. However, the ability to accurately perceive and understand communication signals decreases with age, and it is currently unknown whether face masks impair facial communication more severely in older people. We compared the impact of surgical face masks on dynamic facial emotion recognition in younger (18-30 years) and older (65-85 years) adults (N = 96) in an online study. Participants watched short video clips of young women who facially expressed anger, fear, contempt or sadness. Faces of half of the women were covered by a digitally added surgical face mask. As expected, emotion recognition accuracy declined with age, and face masks reduced emotion recognition accuracy in both younger and older participants. Unexpectedly, the effect of face masks did not differ between age groups. Further analyses showed that masks also reduced the participants' overall confidence in their emotion judgements, but not their performance awareness (the difference between their confidence ratings for correct and incorrect responses). Again, there were no mask-by-age interactions. Finally, data obtained with a newly developed questionnaire (attitudes towards face masks, atom) suggest that younger and older people do not differ in how much they feel impaired in their understanding of other people's emotions by face masks or how useful they find face masks in confining the COVID-19 pandemic. In sum, these findings do not provide evidence that the impact of face masks on the decoding of facial signals is disproportionally larger in older people.

Impact of face masks and sunglasses on emotion recognition in South Koreans.

Due to the prolonged COVID-19 pandemic, wearing masks has become essential for social interaction, disturbing emotion recognition in daily life. In the present study, a total of 39 Korean participants (female = 20, mean age = 24.2 years) inferred seven emotions (happiness, surprise, fear, sadness, disgust, anger, surprise, and neutral) from uncovered, mask-covered, sunglasses-covered faces. The recognition rates were the lowest under mask conditions, followed by the sunglasses and uncovered conditions. In identifying emotions, different emotion types were associated with different areas of the face. Specifically, the mouth was the most critical area for happiness, surprise, sadness, disgust, and anger recognition, but fear was most recognized from the eyes. By simultaneously comparing faces with different parts covered, we were able to more accurately examine the impact of different facial areas on emotion recognition. We discuss the potential cultural differences and the ways in which individuals can cope with communication in which facial expressions are paramount.

Face masks affect perception of happy faces in deaf people.

The SARS-CoV-2 pandemic has led significant social repercussions and forced people to wear face masks. Recent research has demonstrated that the human ability to infer emotions from facial configurations is significantly reduced when face masks are worn. Since the mouth region is specifically crucial for deaf people who speak sign language, the current study assessed the impact of face masks on inferring emotional facial expressions in a population of adult deaf signers. A group of 34 congenitally deaf individuals and 34 normal-hearing individuals were asked to identify happiness, sadness, fear, anger, and neutral expression on static human pictures with and without facial masks presented through smartphones. For each emotion, the percentage of correct responses with and without face masks was calculated and compared between groups. Results indicated that face masks, such as those worn due to the SARS-CoV-2 pandemic, limit the ability of people to infer emotions from facial expressions. The negative impact of face masks is significantly pronounced when deaf people have to recognize low-intensity expressions of happiness. These findings are of essential importance because difficulties in recognizing emotions from facial expressions due to mask wearing may contribute to the communication challenges experienced by the deaf community during the SARS-CoV-2 pandemic, generating feelings of frustration and exclusion.

A neural signature of exposure to masked faces after 18 months of COVID-19.

In the last two years, face-to-face interactions have drastically changed worldwide, because of the COVID-19 pandemic: the persistent use of masks has had the advantage of reducing viral transmission, but it has also had the cost of impacting on the perception and recognition of social information from faces, especially emotions. To assess the cerebral counterpart to this condition, we carried out an EEG experiment, extracting Event-Related Potentials (ERPs) evoked by emotional faces with and without surgical masks. Besides the expected impairment in emotion recognition in both accuracy and response times, also the classical face-related ERPs (N170 and P2) are altered by the presence of surgical masks. Importantly, the effect is stronger in individuals with a lower daily exposure to masks, suggesting that the brain must adapt to an extra constraint in decoding social input, due to masks hiding crucial facial information.

Optimal Compact Network for Micro-Expression Analysis System.

Micro-expression analysis is the study of subtle and fleeting facial expressions that convey genuine human emotions. Since such expressions cannot be controlled, many believe that it is an excellent way to reveal a human's inner thoughts. Analyzing micro-expressions manually is a very time-consuming and complicated task, hence many researchers have incorporated deep learning techniques to produce a more efficient analysis system. However, the insufficient amount of micro-expression data has limited the network's ability to be fully optimized, as overfitting is likely to occur if a deeper network is utilized. In this paper, a complete deep learning-based micro-expression analysis system is introduced that covers the two main components of a general automated system: spotting and recognition, with also an additional element of synthetic data augmentation. For the spotting part, an optimized continuous labeling scheme is introduced to spot the apex frame in a video. Once the apex frames have been recognized, they are passed to the generative adversarial network to produce an additional set of augmented apex frames. Meanwhile, for the recognition part, a novel convolutional neural network, coined as Optimal Compact Network (OC-Net), is introduced for the purpose of emotion recognition. The proposed system achieved the best F1-score of 0.69 in categorizing the emotions with the highest accuracy of 79.14%. In addition, the generated synthetic data used in the training phase also contributed to performance improvement of at least 0.61% for all tested networks. Therefore, the proposed optimized and compact deep learning system is suitable for mobile-based micro-expression analysis to detect the genuine human emotions.

New evidence of impaired expression recognition in developmental prosopagnosia.

Developmental prosopagnosia (DP) is a neurodevelopmental condition characterized by lifelong face recognition difficulties. To date, it remains unclear whether or not individuals with DP experience impaired recognition of facial expressions. It has been proposed that DPs may have sufficient perceptual ability to correctly interpret facial expressions when tasks are relatively easy (e.g., the stimuli are unambiguous and viewing conditions are optimal), but exhibit subtle impairments when tested under more challenging conditions. In the present study, we sought to take advantage of the COVID-19 pandemic to test this view. It is well-established that the surgical-type masks worn during the pandemic hinder the recognition and interpretation of facial emotion in typical participants. Relative to typical participants, we hypothesized that DPs may be disproportionately impaired when asked to interpret the facial emotion of people wearing face masks. We compared the ability of 34 DPs and 60 age-matched typical controls to recognize facial emotions i) when the whole face is visible, and ii) when the lower portion of the face is covered with a surgical mask. When expression stimuli were viewed without a mask, the DPs and typical controls exhibited similar levels of performance. However, when expression stimuli were shown with a mask, the DPs showed signs of subtle expression recognition deficits. The DPs were particularly prone to mislabeling masked expressions of happiness as emotion neutral. These results add to a growing body of evidence that under some conditions, DPs do exhibit subtle deficits of expression recognition.

Anthropomorphic Robotic Eyes: Structural Design and Non-Verbal Communication Effectiveness.

This paper shows the structure of a mechanical system with 9 DOFs for driving robot eyes, as well as the system's ability to produce facial expressions. It consists of three subsystems which enable the motion of the eyeballs, eyelids, and eyebrows independently to the rest of the face. Due to its structure, the mechanical system of the eyeballs is able to reproduce all of the motions human eyes are capable of, which is an important condition for the realization of binocular function of the artificial robot eyes, as well as stereovision. From a kinematic standpoint, the mechanical systems of the eyeballs, eyelids, and eyebrows are highly capable of generating the movements of the human eye. The structure of a control system is proposed with the goal of realizing the desired motion of the output links of the mechanical systems. The success of the mechanical system is also rated on how well it enables the robot to generate non-verbal emotional content, which is why an experiment was conducted. Due to this, the face of the human-like robot MARKO was used, covered with a face mask to aid in focusing the participants on the eye region. The participants evaluated the efficiency of the robot's non-verbal communication, with certain emotions achieving a high rate of recognition.

Face masks have a limited influence on first impressions: evidence from three experiments.

Considering the widespread use of face masks during the COVID-19 pandemic, the goal of the current study was to examine how occlusion of the lower half of the face may impact first impression formation. We conducted three experiments, each building on previous research, investigating the effect of face masks on first impressions of faces across the lifespan (children, young and older adults). Experiment 1 examined whether the mandatory influence of happy facial expressions on perceived trustworthiness in young adult faces is influenced by face masks. Experiment 2 examined behavioural consequences of adults' first impressions of child faces to determine whether masks reduce the effect of facial niceness on interpretations of ambiguous behaviour. Experiment 3 investigated consensus for first impressions of trustworthiness and competence in older adult faces with and without masks, as well as consensus on underlying facial cues. The results of all three experiments present converging evidence that masks do not have a significant impact on first impressions and their behavioural consequences.

The Effect of Surgical Masks on the Featural and Configural Processing of Emotions.

From the start of the COVID-19 pandemic, the use of surgical masks became widespread. However, they occlude an important part of the face and make it difficult to decode and interpret other people's emotions. To clarify the effect of surgical masks on configural and featural processing, participants completed a facial emotion recognition task to discriminate between happy, sad, angry, and neutral faces. Stimuli included fully visible faces, masked faces, and a cropped photo of the eyes or mouth region. Occlusion due to the surgical mask affects emotion recognition for sadness, anger, and neutral faces, although no significative differences were found in happiness recognition. Our findings suggest that happiness is recognized predominantly via featural processing.

Memory advantage for untrustworthy faces: Replication across lab- and web-based studies.

The Covid-19 pandemic imposed new constraints on empirical research and forced researchers to transfer from traditional laboratory research to the online environment. This study tested the validity of a web-based episodic memory paradigm by comparing participants' memory performance for trustworthy and untrustworthy facial stimuli in a supervised laboratory setting and an unsupervised web setting. Consistent with previous results, we observed enhanced episodic memory for untrustworthy compared to trustworthy faces. Most importantly, this memory bias was comparable in the online and the laboratory experiment, suggesting that web-based procedures are a promising tool for memory research.

Face masks have emotion-dependent dissociable effects on accuracy and confidence in identifying facial expressions of emotion.

The coronavirus pandemic has resulted in increased use of face masks worldwide. Here, we examined the effect of wearing a face mask on the ability to recognise facial expressions of emotion. In a within-subjects design, 100 UK-based undergraduate students were shown facial expressions of anger, disgust, fear, happiness, sadness, and neutral expression; these were either posed with or without a face mask, or with a face mask artificially imposed onto them. Participants identified the emotion portrayed in the photographs from a fixed choice array of answers and rated their confidence in their selection. While overall accuracy was higher without than with masks, the effect varied across emotions, with a clear advantage without masks in disgust, happiness, and sadness; no effect for neutral, and lower accuracy without masks for anger and fear. In contrast, confidence was generally higher without masks, with the effect clear for all emotions other than anger. These results confirm that emotion recognition is affected by face mask wearing, but reveal that the effect depends on the emotion being displayed-with this emotion-dependence not reflected in subjects' confidence. The disparity between the effects of mask wearing on different emotions and the failure of this to be reflected in confidence ratings suggests that mask wearing not only effects emotion recognition, but may also create biases in the perception of facial expressions of emotion of which perceivers are unaware. In addition, the similarity of results between the Imposed Mask and Posed Mask conditions suggests that prior research using artificially imposed masks has not been deleteriously affected by the use of this manipulation.

How does the presence of a surgical face mask impair the perceived intensity of facial emotions?

The use of surgical-type face masks has become increasingly common during the COVID-19 pandemic. Recent findings suggest that it is harder to categorise the facial expressions of masked faces, than of unmasked faces. To date, studies of the effects of mask-wearing on emotion recognition have used categorisation paradigms: authors have presented facial expression stimuli and examined participants' ability to attach the correct label (e.g., happiness, disgust). While the ability to categorise particular expressions is important, this approach overlooks the fact that expression intensity is also informative during social interaction. For example, when predicting an interactant's future behaviour, it is useful to know whether they are slightly fearful or terrified, contented or very happy, slightly annoyed or angry. Moreover, because categorisation paradigms force observers to pick a single label to describe their percept, any additional dimensionality within observers' interpretation is lost. In the present study, we adopted a complementary emotion-intensity rating paradigm to study the effects of mask-wearing on expression interpretation. In an online experiment with 120 participants (82 female), we investigated how the presence of face masks affects the perceived emotional profile of prototypical expressions of happiness, sadness, anger, fear, disgust, and surprise. For each of these facial expressions, we measured the perceived intensity of all six emotions. We found that the perceived intensity of intended emotions (i.e., the emotion that the actor intended to convey) was reduced by the presence of a mask for all expressions except for anger. Additionally, when viewing all expressions except surprise, masks increased the perceived intensity of non-intended emotions (i.e., emotions that the actor did not intend to convey). Intensity ratings were unaffected by presentation duration (500ms vs 3000ms), or attitudes towards mask wearing. These findings shed light on the ambiguity that arises when interpreting the facial expressions of masked faces.